Method and apparatus for processing image based on optical communication, and computer-readable recording medium with program therefor

ABSTRACT

An optical communication-based image processing system is disclosed. The system may include a transmitter having at least one light emitting element and a receiver having a rolling shutter camera. As the system is provided, a rolling shutter effect may be improved.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. 119(a) to KoreanPatent Application No. 10-2020-0185073, filed on Dec. 28, 2020, which isincorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present disclosure relates to an optical communication technologyusing a light emitting element as a transmitter and a camera having animage sensor as a receiver, and a technology of processing a receivedimage based on optical communication. The present invention resultedfrom “Intelligent Internet of Energy (IoE) Data Research” of “UniversityICT Research Center Support Project” supported by the Ministry ofScience and ICT (Project No.: 1711116158)

Description of the Related Art

Visible light communication (VLC), a representative lightingcommunication convergence technology, is a technology of performingwireless communication by loading information in light from a lightsource. Conventionally, light of a light source is received by aphotodiode (PD), digital data 1 or 0 is detected according to on/off ofthe light source, and information is transmitted by combining thedigital data.

Conventionally, a VLC system has been proposed to capture a plurality oflight emitting diodes (LEDs) by a camera, instead of a photodiode, andextract data corresponding to on/off of the LEDs obtained on a framebasis in the camera. The VLC system using a camera is also called anoptical camera communication (OCC) system in that it uses a camerainstead of a photodiode as an optical receiver, and the IEEE 802.15.7atask group is attempting to standardize the OCC system.

An attempt has recently been made to apply a rolling shutter camera asthe camera of the OCC system. The rolling shutter camera obtains animage on a frame basis by capturing an on/off image of LEDs in each rowin image sensors arranged in a plurality of rows.

However, when moving LEDs are captured by the rolling shutter camera, anon/off image is captured on a row basis, thereby creating a step-shapeddistorted image. Accordingly, there is a need for a method of overcomingthe problem.

The above information is presented as background information only toassist with an understanding of the present invention. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present invention.

SUMMARY OF THE INVENTION

Therefore, the present invention has been made in view of the aboveproblems, and it is an object of the present invention to provide animage processing system which corrects an image experiencing a rollingshutter phenomenon.

It is another object of the present invention to provide an opticalcommunication-based image processing system which provides aflicker-free environment, while transmitting encoded data in differentfrequencies.

It is a further object of the present invention to provide an opticalcommunication-based image processing system which reduces a bit errorrate (BER) by using an image correction model.

In accordance with the present invention, the above and other objectscan be accomplished by the provision of an optical communication-basedimage processing apparatus including a rolling shutter camera configuredto capture an image of at least one light emitting element, when sensingdata collected from at least one measurement sensor is encoded andtransmitted through the at least one light emitting element, and areception controller configured to input the captured light emittingelement image to a pre-trained image correction model.

The image correction model may be configured to output a light emittingelement image captured at a capturing time or a light emitting elementimage with a corrected distortion area, based on whether the rollingshutter camera has moved relative to the at least one light emittingelement at the capturing time.

The reception controller may be configured to decode the encoded sensingdata based on the output light emitting element image captured at thecapturing time or the output light emitting element image with thecorrected distortion area.

According to an embodiment, an optical communication-based imageprocessing method includes capturing an image of at least one movinglight emitting element by using a rolling shutter camera, when sensingdata collected from at least one measurement sensor is encoded andtransmitted through the at least one moving light emitting element,inputting the captured light emitting element image to a pre-trainedimage correction model, outputting a light emitting element imagecaptured at a capturing time or a light emitting element image with acorrected distortion area from the image correction model, based onwhether the rolling shutter camera has moved relative to the at leastone light emitting element at the capturing time, and decoding theencoded sensing data based on the output light emitting element imagecaptured at the capturing time or the output light emitting elementimage with the corrected distortion area.

Further, an optical communication-based image processing methodaccording to an embodiment may be stored in a computer-readablerecording medium recording a program to be executed on a computer.

The program may include an executable instruction which when executed bya processor, causes the processor to perform capturing an image of atleast one moving light emitting element by using a rolling shuttercamera, when sensing data collected from at least one measurement sensoris encoded and transmitted through the at least one moving lightemitting element, inputting the captured light emitting element image toa pre-trained image correction model, outputting a light emittingelement image captured at a capturing time or a light emitting elementimage with a corrected distortion area from the image correction model,based on whether the rolling shutter camera has moved relative to the atleast one light emitting element at the capturing time, and decoding theencoded sensing data based on the output light emitting element imagecaptured at the capturing time or the output light emitting elementimage with the corrected distortion area.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and other advantages of thepresent invention will be more clearly understood from the followingdetailed description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram illustrating an optical communication-based imageprocessing system according to an embodiment of the present invention;

FIG. 2 is a system block diagram illustrating the configuration of anoptical communication-based image processing system according to anembodiment of the present invention;

FIG. 3 is a diagram illustrating the operation principle of a rollingshutter camera according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating an operation of an opticalcommunication-based transmitter, for encoding sensing data according toan embodiment of the present invention;

FIGS. 5A to 5C are diagrams referred to for describing a rolling shuttercamera that captures light emitting elements in different frequenciesaccording to an embodiment of the present invention;

FIG. 6 is a diagram illustrating a process of training an imagecorrection model according to an embodiment of the present invention;

FIG. 7 is a flowchart illustrating a process of applying a pre-trainedimage correction model according to an embodiment of the presentinvention;

FIGS. 8A to 8D are diagrams illustrating a process of capturing imagesof light emitting elements, recognizing the captured images, andprocessing the images according to an embodiment of the presentinvention;

FIG. 9A is a graph illustrating a normalized intensity corresponding toa row index according to an embodiment of the present invention and FIG.9B is a graph illustrating the number of bits recognized according to adistance;

FIG. 10A is a graph illustrating a bit error rate (BER) according to arelative velocity according to an embodiment of the present invention,and FIG. 10B is a graph illustrating a BER corresponding to the totalnumber of epochs according to an embodiment of the present invention;

FIG. 11A, FIG. 11B, and FIG. 11C are diagrams illustrating a rollingshutter phenomenon in the case of using N×N light emitting elements, andFIG. 11D is a diagram illustrating a data rate according to a distance;and

FIG. 12 is a flowchart illustrating an operation of an opticalcommunication-based image processing system 1000 according to anembodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to exemplary embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings. The present invention may be implemented in various forms, andshould not be interpreted as being limited to the embodiments describedherein. In the drawings, the relative sizes of components, layers, andareas may be exaggerated, for clarity of description.

FIG. 1 is a diagram illustrating an optical communication-based imageprocessing system 1000 according to an embodiment of the presentinvention.

The optical communication-based image processing system 1000 may includea transmitter 100 that optically transmits data and a receiver 200 thatincludes or communicates with a rolling shutter camera 210 having animage sensor. The receiver 200 may be implemented as an image processingdevice 200.

The transmitter 100 may optically output sensing data collected from oneor more measurement sensors through one or more light emitting elements130. The measurement sensors may include a temperature sensor and/or ahumidity sensor, to which an embodiment is not limited.

There may be a plurality of light emitting elements 130 which may bearranged in a 4×4 array, an 8×8 array, or the like. Depending on animplementation, a single light emitting element may be configured. In anembodiment, the number of light emitting elements 130 may be changedbased on the type of measurement sensors and the amount of data.

The plurality of light emitting elements 130 may optically transmitencoded sensing data collected from the measurement sensors. Forexample, the plurality of light emitting elements 130 may output apredetermined property value (e.g., a brightness value) includingsymbols. In an alternative embodiment, variable pulse width modulation(VPWM), which is a modulation scheme with a variable pulse width offset,may be applied as an encoding method.

The rolling shutter camera 210 may capture the plurality of lightemitting elements 130, while moving from a first position 210 a to asecond position 210 b. Because the rolling shutter camera 210 scans aframe row by row or column by column rather than a full frame at once asis done with a global shutter method, a geometric distortion called arolling shutter phenomenon (or jello effect) appears in a capturedimage.

In an alternative or additional embodiment, even when the rollingshutter camera 210 in a fixed state captures the plurality of lightemitting elements 130, the rolling shutter phenomenon may occur.

In an embodiment, the rolling shutter camera 210 may create the rollingshutter phenomenon depending on whether the rolling shutter camera 210moves relative to the plurality of light emitting elements 130 at acapturing time. In addition, since the plurality of light emittingelements 130 are repeatedly on and off, a bright stripe and a darkstripe may alternate in an image captured by the rolling shutter camera210. Herein, a relative movement refers to the movement of the rollingshutter camera 210 during capturing or the movement of the plurality oflight emitting elements 130 during capturing, while the rolling shuttercamera 210 is fixed. In the present disclosure, movement mainly meansthe above-described relative movement.

The image processing system 1000 according to an embodiment of thepresent invention may correct an image experiencing the rolling shutterphenomenon. Accordingly, even though a separate device (e.g., a globalshutter camera) is not applied, an image free of the rolling shutterphenomenon may be output.

FIG. 2 is a system block diagram illustrating the configuration of theoptical communication-based image processing system 1000 according to anembodiment of the present invention. Since each of the componentsillustrated in FIG. 2 is not compulsory for implementing the imageprocessing system 1000, more or fewer components may be included in theimage processing system 1000.

The image processing system 1000 may include the opticalcommunication-based transmitter 100 and receiver 200. As describedabove, the receiver 200 may be used interchangeably with the imageprocessing device 200.

The transmitter 100 is a module that transmits data and includes ameasurement sensor 110, a light emitting element 130, and a transmissioncontroller 190.

The measurement sensor 110 may include one or more measurement sensorsthat measure environmental data. Specifically, the measurement sensor110 may include a temperature sensor 111 and a humidity sensor 113.However, as far as optical communication is possible, the measurementsensor 110 may include sensors that measure various environmental data,biometric data, spatial data, and so on.

One or more light emitting elements 130 may be implemented, and thenumber of light emitting elements 130 may be determined according to theamount of collected sensing data. The present invention will bedescribed in the context of a plurality of emitting elements 130.

Each individual one of the light emitting elements 130 may be configuredwith an outer diameter of 3 millimeters, to which an embodiment is notlimited.

The transmission controller 190 may encode sensing data collected fromthe measurement sensor 110 and transmit the encoded sensing data throughthe light emitting elements 130.

The transmission controller 190 may encode the sensing data collectedfrom the measurement sensor based on VPWM.

The transmission controller 190 may generate the encoded data as a datapacket and transmit the data packet through the light emitting elements130. VPWM may configure a different LED on/off ratio for each pulse,like pulse width modulation (PWM).

The receiver 200 may include the rolling shutter camera 210, a memory240 storing an image correction model 241, and a reception controller290.

The rolling shutter camera 210 may capture the light emitting elements130, while moving. In an alternative embodiment, the rolling shuttercamera 210 in a fixed state may capture the light emitting elements 130which are moving or fixed.

The memory 240 may store various types of information and the imagecorrection model 241 which has already been trained. The pre-trainedimage correction model 241 may be implemented based on supervisedlearning, to which an embodiment is not limited.

The image correction model 241 may recognize the plurality of lightemitting elements 130. That is, the image correction model 241 mayrecognize the number and arrangement direction of the light emittingelements 130. The image correction model 241 may be trained to correct adistortion area with the rolling shutter phenomenon in an image of thelight emitting elements 130. In addition, the image correction model 241may use a specific light emitting element as a reference light emittingelement in order to recognize the arrangement direction of the pluralityof light emitting elements 130.

An operation of the image correction model 241 according to variousembodiments of the present invention will be described in detail withreference to FIGS. 6 and 7 . In the following description, the referencenumerals of FIG. 2 will also be referred to.

FIG. 3 is a diagram illustrating an operation principle of the rollingshutter camera 210 according to an embodiment of the present invention.

The rolling shutter camera 210 may capture the plurality of lightemitting elements 130 at a predetermined frame speed (e.g., 30 fps). Therolling shutter camera 210 may capture the plurality of light emittingelements 130 for each of frames Fr1, Fr2, . . . .

A shutting operation refers to blocking light at an image sensor, andthe image sensor provided in the rolling shutter camera 210 may react tolight, generate charge when receiving light, performs charge reset RT1in an unnecessary period, and output the charge of a necessary period byreadout ROT1. A time period from the reset RT1 to the readout ROT1 isreferred to as an exposure time.

The brightness of a captured image may be determined according to theexposure time, and the intensities of a bright stripe and a dark stripeof the light emitting elements 130 may be determined according to theexposure time.

The rolling shutter camera 210 does not recognize a full frame for eachof the frames Fr1 and Fr2. Rather, the rolling shutter camera 210outputs charges accumulated in pixels column by column or row by row(e.g., from a first row to an n^(th) row). Therefore, when the rollingshutter camera 210 captures the plurality of light emitting elements130, while being shaken, the rolling shutter camera 210 may output animage including a distortion area.

FIG. 4 is a diagram illustrating an operation of the opticalcommunication-based transmitter 100, for encoding sensing data accordingto an embodiment of the present invention.

The transmission controller 190 of the transmitter 100 may encodesensing data collected from the measurement sensor 110 based on VPWM.

Specifically, the transmission controller 190 may encode data for eachof data packets DP1 and DP2 and transmit the encoded data. Each of thedata packets DP1 and DP2 may be generated in the order of header data,core data, and footer data. The header data and the footer data mayinclude a series of data bits distinguished from the core data. Forexample, the header data may include “11100”, and the footer data mayinclude “10100”, to which an embodiment is not limited.

The transmission controller 190 may delete one or more pulses having aduty cycle of 0% or 100% included in a data packet. Accordingly, aflicker phenomenon may be prevented from occurring in a period havingconsecutive duty cycles.

The transmission controller 190 may use a 50% duty cycle as “0” and a75% duty cycle as “1”. To exclude continuity of “0” or “1”, thetransmission controller 190 may fix the pulse width of an off part andvariably adjust the pulse width of an on part, for a duty cycle at apredetermined position included in the data packet. Accordingly, theflicker phenomenon may be prevented in a period having consecutive dutycycles.

FIGS. 5A to 5C are diagrams referred to for describing the rollingshutter camera 210 that captures light emitting elements in differentfrequencies according to an embodiment of the present invention.

The rolling shutter camera 210 may capture the plurality of lightemitting elements 130 by setting a resolution of 640×480, setting 500 Hz(FIG. 5A), 1000 Hz (FIG. 5B), and 2000 Hz (FIG. 5C), and settingexposure times of 0.65 ms, 1.25 ms, and 2.5 ms, to which an embodimentis not limited.

Referring to FIGS. 5A to 5C, it may be observed that the width of brightstripes increases as the frequency decreases (STa to STc).

The receiver 200 may perform Gaussian blur processing, gray scale formatprocessing for removing a background, and the like on an image capturedby the rolling shutter camera 210.

FIG. 6 is a diagram illustrating a process of training the imagecorrection model 241 according to an embodiment of the presentinvention, and FIG. 7 is a diagram illustrating a process of applyingthe pre-trained image correction model 241 according to an embodiment ofthe present invention.

Referring to FIG. 6 , the image correction model 241 may receive a lightemitting element image captured by the rolling shutter camera 210 in atraining step Tr. The light emitting element image may be an image 610in which the rolling shutter phenomenon has occurred or an image 620A inwhich the rolling shutter phenomenon has not occurred.

The image correction model 241 may receive patches 610 a to 610 k and620 a to 620 k of the light emitting element image, and may alreadystore or receive one or more reference light emitting element images (orpatch images). A reference light emitting element image may be a lightemitting element image captured while all of the camera and the lightemitting elements are stationary, and may be used as a reference fordetermining a relative movement of the captured light emitting elementsand the accuracy of distortion correction.

The image correction model 241 may determine whether the rolling shuttercamera 210 has moved relative to training light emitting elements at acapturing time, based on whether the edge shapes of adjacent stripes areuniform in the stripe pattern of the captured image of the traininglight emitting elements. In an embodiment, instead of the imagecorrection model 241 controlled by the reception controller 290, thereception controller 290 may directly make the determination as towhether the rolling shutter camera 210 has made a relative movement.

The image correction model 241 may be trained to correct the distortionarea of the image of the training light emitting elements, with respectto which the rolling shutter camera 210 has made a relative movement,based on the reference light emitting element image. The distortion areamay include an area having the rolling shutter phenomenon (including anarea in which stripes intersect with each other).

The image correction model 241 outputs a corrected image 640A andcompares the output image 640A with a reference light emitting elementimage 630. When the similarity between the output image 640A and thereference light emitting element image 630 is less than a predeterminedthreshold, a parameter (a weight, a bias, or the like) of the imagecorrection model 241 may be updated so that the output image 640A of theimage correction model 241 becomes more similar to the reference lightemitting element image 630 (backpropagation). In an embodiment, theparameter may be updated regardless of similarity. In an embodiment, theimage correction model 241 may output information about the coordinatedifference between an input image and an output image, withoutoutputting the image.

When additional correction is required for the light emitting elementimage 640A in which the distortion area has been corrected, thereception controller 290 may perform spatial transformation and imageresampling. In an embodiment, the reception controller 290 may compare aresampled light emitting element image 640B with the reference lightemitting element image 630. When the similarity between the resampledlight emitting element image 640B and the reference light emittingelement image 630 is less than a predetermined threshold, the receptioncontroller 290 may update the parameter of the image correction model241.

The similarity may be obtained by comparing the positions of edges ofstripe patterns in the captured light emitting element image and thereference light emitting element image, and the predetermined thresholdmay be, but not limited to, for example, 85%.

Referring to FIG. 7 , when sensing data collected from one or moremeasurement sensors is encoded and transmitted through one or more lightemitting elements in an application step APP, the image correction model241 which has been completely trained receives a light emitting elementimage Im1 captured by the rolling shutter camera 210 in operation SS0.

The image correction model 241 may determine whether the rolling shuttercamera 210 has moved relative to the light emitting elements at acapturing time in operation NN1 a. Specifically, the image correctionmodel 241 may determine whether the rolling shutter camera 210 has movedrelative to the light emitting elements at the capturing time based onwhether the edge shapes of adjacent stripes in a stripe pattern of thecaptured light emitting element image are uniform under the control ofthe reception controller 290.

When there is no relative movement in operation NN1 a, the imagecorrection model 241 outputs the captured light emitting element imageas it is input, and the reception controller 290 may decode the encodedsensing data based on the captured light emitting element image inoperation SS3.

When there is a relative movement in operation NN1 a, the imagecorrection model 241 may correct a distortion area of a training imageof light emitting elements, relative to which the rolling shutter camera210 has moved, based on a reference light emitting element image inoperation NN1 b. As described above, the distortion area refers to anarea in which the edge shapes of stripes are not uniform.

The image correction model 241 may output the light emitting elementimage with the corrected distortion area in operation SS1, and thereception controller 290 may decode the encoded sensing data based onthe corrected light emitting element image.

The image correction model 241 described above may include a neuralnetwork-based regression algorithm (e.g., one or more convolutionalneural networks (CNNs)). In addition, the above-described referencelight emitting element image or coordinate information corresponding tothe reference light emitting element image may be used as a label of theneural network algorithm. In addition, the light emitting element imagemay be an image obtained by capturing one light emitting element, andmay be an image obtained by capturing a plurality of light emittingelements.

FIGS. 8A to 8D are diagrams illustrating a process of capturing imagesof light emitting elements, recognizing the captured images, andprocessing the images according to an embodiment of the presentinvention.

The receiver 200 may determine the number, arrangement direction, andsize (width and height) of light emitting elements 710 captured by therolling shutter camera 210 (FIG. 8A and FIG. 8B).

Then, the receiver 200 may cancel generated interference or noise toperform filter a stripe pattern (FIG. 8C and FIG. 8D).

In addition to a light emitting element transmitting an optical signal,a light emitting area resulting from light reflection may be observed inthe captured light emitting element image, as illustrated in FIG. 8C.Although the reception controller 290 of the receiver 200 may detect thelight emitting area in the captured image by using a region of Interest(ROI) detection algorithm, the reception controller 290 may determine alight-reflected area (also observed in FIG. 11A and FIG. 11B) as an ROI,in addition to an area in which an actual optical signal is transmitted.

The reception controller 290 may more accurately identify the area inwhich an actual optical signal is transmitted by an optical signal areadetection algorithm that filters out this light-reflected area.

The optical signal area detection algorithm may be configured to detectat least one light emitting area in a captured image, and filter out thelight emitting area, when a stripe pattern of the light emitting areadoes not change in consecutive image frames.

This relies on the property that in the case of an optical signaltransmitted in a high frequency without the flickering effect, althougha stripe pattern appears in one frame captured by the rolling shuttercamera and changes in the next frame, such a stripe pattern change isnot observed in reflected light.

When there are a plurality of light emitting elements, the receptioncontroller 290 of the receiver 200 may control the rolling shuttercamera 210 with a predetermined exposure time in which the plurality oflight emitting elements 130 may be recognized.

In order to prevent interference between light emitting elements, thereceiver 200 may set a small exposure time that enables light emittingelements to be recognized. When a plurality of light emitting elements130 are cognized over time, the receiver may gradually increase theexposure time of the rolling shutter camera over time.

FIG. 9A is a graph illustrating a normalized intensity corresponding toa row index according to an embodiment of the present invention, andFIG. 9B is a graph illustrating the number of recognized bits accordingto a distance.

Referring to FIG. 9A, as the row index increases, the intensity appearsin a stable pattern. Referring to FIG. 9B, as the distance increases,the number of recognized bits may decrease. The number of bits in a lowfrequency may be smaller than the number of bits in a high frequency.

FIG. 10A illustrates a bit error rate (BER) according to a relativevelocity according to an embodiment of the present invention, and FIG.10B illustrates a BER corresponding to the total number of epochsaccording to an embodiment of the present invention.

Once an image correction model is applied, the BER is significantlyreduced. Although a change of the BER according to an exposure time isnot significant, the BER may be smaller as the exposure time is shorter.

FIG. 11A, FIG. 11B, and FIG. 11C are diagrams illustrating the rollingshutter phenomenon in the case of N×N light emitting elements, and FIG.11D illustrates a data rate according to a distance. As the number oflight emitting elements increases, the data rate naturally increases. Inthe case of 8×8 light emitting elements, up to 64 kbps may be observed.

FIG. 12 is a flowchart illustrating an operation of the opticalcommunication-based image processing system 1000 according to anembodiment of the present invention.

First, the transmitter 100 collects environmental data from measurementsensors in operation 51110, encodes the sensing data in operation 51120,and transmits the encoded sensing data through moving light emittingelements in operation 51130.

Upon receipt of sensing data which has been collected from one or moremeasurement sensors and encoded through one or more light emittingelements, the receiver 200 may capture moving light emitting elements bythe rolling shutter camera in operation 51140, select a light emittingelement area that transmits an optical signal in the image, and correcta distortion area in the captured light emitting element image inoperation 51150. In this case, the receiver 200 may output the correctedimage in operation S1160.

The optical signal area detection algorithm may be configured to detectat least one light emitting area in a captured image, and filter out thelight emitting area, when a stripe pattern of the light emitting area isnot changed in consecutive image frames.

More specifically, an optical communication-based image processingmethod may include capturing an image of at least one moving lightemitting element by using a rolling shutter camera, when sensing datacollected from at least one measurement sensor is encoded andtransmitted through the at least one moving light emitting element,inputting the captured light emitting element image to a pre-trainedimage correction model, and when a light emitting element image capturedat a capturing time or a light emitting element image with a correcteddistortion area is output from the image correction model, based onwhether the rolling shutter camera has moved relative to the at leastone light emitting element at the capturing time, decoding the encodedsensing data based on the output light emitting element image capturedat the capturing time or the output light emitting element image withthe corrected distortion area.

Before receiving the light emitting element image in the applicationstep, the receiver 200 may train the image correction mode.

The training may include receiving at least one training light emittingelement image obtained by capturing at least one training light elementthrough the rolling shutter camera and at least one reference lightemitting element image, determining whether the rolling shutter camerahas moved relative to the at least one training light emitting elementat the capturing time, based on whether the shapes of edges of adjacentstripes in a stripe pattern of the captured training light emittingelement image are uniform, and correcting a distortion area of thetraining light emitting element image captured by the rolling shuttercamera which has moved relatively, based on the reference light emittingelement image.

The training may further include, when a similarity between the trainingmoving light emitting element image with the corrected distortion areaand the reference light emitting element image is less than apredetermined threshold, updating a parameter of the image correctionmodel is updated.

The above-described present invention may be implemented ascomputer-readable code in a medium to which a program is recorded. Thecomputer-readable recording medium includes any type of recording devicein which data readable by a computer system is stored. Examples of thecomputer-readable medium includes a hard disk drive (HDD), a solid statedisk (SSD), a silicon disk drive (SDD), a read only memory (ROM), arandom access memory (RAM), a compact disk-ROM (CD-ROM), a magnetictape, a floppy disk, and an optical data storage device. Further, thecomputer may include the reception controller 290 of the receiver 200.

Specifically, the program may include an executable instruction whichwhen executed by a processor, causes the processor to perform capturingan image of at least one moving light emitting element by using arolling shutter camera, when sensing data collected from at least onemeasurement sensor is encoded and transmitted through the at least onemoving light emitting element, inputting the captured light emittingelement image to a pre-trained image correction model, outputting alight emitting element image captured at a capturing time or a lightemitting element image with a corrected distortion area from the imagecorrection model, based on whether the rolling shutter camera has movedrelative to the at least one light emitting element at the capturingtime, and decoding the encoded sensing data based on the output lightemitting element image captured at the capturing time or the outputlight emitting element image with the corrected distortion area.

The processor may further perform training the image correction mode,before the inputting, which has been described before and thus will notbe described herein.

As is apparent from the above description, according to variousembodiments, the rolling shutter phenomenon may be corrected simply byimage processing without applying a separate device (e.g., a globalshutter camera), and flicker-free data transmission and reception may beperformed regardless of a frequency.

While a specific embodiment of the present invention has been describedand illustrated above, the present invention is not limited to thedescribed embodiment. Those skilled in the art will understand that manymodifications and variations can be made without departing from thescope and spirit of the present invention to achieve other specificembodiments.

What is claimed is:
 1. An optical communication-based image processingapparatus comprising: a rolling shutter camera configured to capture animage of at least one light emitting element, when sensing datacollected from at least one measurement sensor is encoded andtransmitted through the at least one light emitting element; and areception controller configured to input the captured light emittingelement image to a pre-trained image correction model, wherein the imagecorrection model is configured to output a light emitting element imagecaptured at a capturing time or a light emitting element image with acorrected distortion area, based on whether the rolling shutter camerahas moved relative to the at least one light emitting element at thecapturing time, wherein the reception controller is configured to decodethe encoded sensing data based on the output light emitting elementimage captured at the capturing time or the output light emittingelement image with the corrected distortion area, and wherein the imagecorrection model is configured to: receive at least one training lightemitting element image obtained by capturing at least one training lightelement through the rolling shutter camera and at least one referencelight emitting element image; determine whether the rolling shuttercamera has moved relative to the at least one training light element atthe capturing time, based on whether the shapes of edges of adjacentstripes in a stripe pattern of the captured training light emittingelement image are uniform; and be trained to correct a distortion areaof the training light emitting element image captured by the rollingshutter camera which has moved relatively, based on the reference lightemitting element image.
 2. The optical communication-based imageprocessing apparatus according to claim 1, wherein when a similaritybetween the training light emitting element image with the correcteddistortion area and the reference light emitting element image is lessthan a predetermined threshold, a parameter of the image correctionmodel is updated.
 3. The optical communication-based image processingapparatus according to claim 1, wherein the reception controller isconfigured to perform spatial transformation and resampling on thetraining light emitting element image with the corrected distortionarea, for additional correction.
 4. The optical communication-basedimage processing apparatus according to claim 1, wherein the sensingdata collected from the at least one measurement sensor is encoded basedon Variable Pulse Width Modulation (VPWM) and transmitted through the atleast one light emitting element.
 5. The optical communication-basedimage processing apparatus according to claim 4, wherein the encodedsensing data is generated as a data packet, wherein header data, coredata, and footer data are sequentially arranged in the data packet, andwherein each of the header data and the footer data includes a series ofdata bits distinguished from the core data.
 6. The opticalcommunication-based image processing apparatus according to claim 5,wherein at least one pulse having a duty cycle of 0% or 100% included inthe data packet is deleted, and wherein a pulse width of an off part isfixed and a pulse width of an on part is variably adjusted, for a dutycycle at a predetermined position in the data packet.
 7. The opticalcommunication-based image processing apparatus according to claim 1,wherein the at least one measurement sensor includes a temperaturesensor and a humidity sensor, and wherein the number of light emittingelements that will be used for communication is determined based on theamount of data collected by the at least one measurement sensor.
 8. Theoptical communication-based image processing apparatus according toclaim 1, wherein when a plurality of light emitting elements exist, thereception controller is configured to control the rolling shutter camerato a predetermined exposure time during which the plurality of lightemitting elements are recognizable, and wherein when the plurality oflight emitting elements are recognized, the reception controller isconfigured to gradually increase the exposure time of the rollingshutter camera over time.
 9. The optical communication-based imageprocessing apparatus according to claim 1, wherein the receptioncontroller is configured to detect at least one light emitting area inthe captured image, and when a stripe pattern of the light emitting areais not changed in consecutive image frames, filter out the lightemitting area.
 10. The optical communication-based image processingapparatus according to claim 1, wherein the reception controller isconfigured to: determine whether the rolling shutter camera has movedrelative to the at least one light emitting element at the capturingtime, based on whether the shapes of edges of adjacent stripes in astripe pattern of the captured light emitting element image are uniform;when the absence of a relative movement of the light emitting element isdetermined, decode the encoded sensing data based on the captured lightemitting element image; and when the presence of a relative movement ofthe light emitting element is determined, decode the encoded sensingdata based on a light emitting element image obtained by correcting adistortion area of the light emitting element image according to theimage correction model trained based on a reference light emittingelement image for the captured light emitting element.
 11. An opticalcommunication-based image processing method comprising: capturing animage of at least one moving light emitting element by using a rollingshutter camera, when sensing data collected from at least onemeasurement sensor is encoded and transmitted through the at least onemoving light emitting element; inputting the captured light emittingelement image to a pre-trained image correction model; outputting alight emitting element image captured at a capturing time or a lightemitting element image with a corrected distortion area from the imagecorrection model, based on whether the rolling shutter camera has movedrelative to the at least one light emitting element at the capturingtime; and decoding the encoded sensing data based on the output lightemitting element image captured at the capturing time or the outputlight emitting element image with the corrected distortion area,training the image correction mode, before the inputting, wherein thetraining comprises: receiving at least one training light emittingelement image obtained by capturing at least one training light elementthrough the rolling shutter camera and at least one reference lightemitting element image; determining whether the rolling shutter camerahas moved relative to the at least one training light emitting elementat the capturing time, based on whether the shapes of edges of adjacentstripes in a stripe pattern of the captured training light emittingelement image are uniform; and correcting a distortion area of thetraining light emitting element image captured by the rolling shuttercamera which has moved relatively, based on the reference light emittingelement image.
 12. The optical communication-based image processingmethod according to claim 11, wherein the training further comprises,when a similarity between the training moving light emitting elementimage with the corrected distortion area and the reference lightemitting element image is less than a predetermined threshold, updatinga parameter of the image correction model is updated.
 13. The opticalcommunication-based image processing method according to claim 11,wherein the sensing data collected from the at least one measurementsensor is encoded based on Variable Pulse Width Modulation (VPWM). 14.The optical communication-based image processing method according toclaim 13, wherein the encoded sensing data is transmitted as a datapacket, wherein header data, core data, and footer data are sequentiallyarranged in the data packet, and wherein each of the header data and thefooter data includes a series of data bits distinguished from the coredata.
 15. The optical communication-based image processing methodaccording to claim 14, wherein at least one pulse having a duty cycle of0% or 100% included in the data packet is deleted, and wherein a widthof an off part is maintained and a width of an on part is variablyadjusted, for a duty cycle at a predetermined position in the datapacket.
 16. The optical communication-based image processing methodaccording to claim 11, wherein the at least one measurement sensorincludes a temperature sensor and a humidity sensor, and wherein thenumber of light emitting elements that will be used for communication isdetermined based on the amount of data collected by the at least onemeasurement sensor.
 17. A computer-readable recording medium recording aprogram to be executed on a computer, wherein the program includes anexecutable instruction which when executed by a processor, causes theprocessor to perform: capturing an image of at least one moving lightemitting element by using a rolling shutter camera, when sensing datacollected from at least one measurement sensor is encoded andtransmitted through the at least one moving light emitting element;inputting the captured light emitting element image to a pre-trainedimage correction model; outputting a light emitting element imagecaptured at a capturing time or a light emitting element image with acorrected distortion area from the image correction model, based onwhether the rolling shutter camera has moved relative to the at leastone light emitting element at the capturing time; and decoding theencoded sensing data based on the output light emitting element imagecaptured at the capturing time or the output light emitting elementimage with the corrected distortion area, wherein the program furtherincludes an executable instruction which causes the processor to trainthe image correction model, before the inputting, wherein the trainingcomprises: receiving at least one training light emitting element imageobtained by capturing at least one training light element through therolling shutter camera and at least one reference light emitting elementimage; determining whether the rolling shutter camera has moved relativeto the at least one training light emitting element at the capturingtime, based on whether the shapes of edges of adjacent stripes in astripe pattern of the captured training light emitting element image areuniform; and correcting a distortion area of the training light emittingelement image captured by the rolling shutter camera which has movedrelatively, based on the reference light emitting element image.